docs: add release notes and docs for batched shuffle insertion (PR #1369) - #1698
Conversation
Greptile SummaryThis PR adds release notes and documentation for the batched shuffle insertion feature (
Confidence Score: 5/5Documentation-only PR with no code changes; safe to merge. All findings are P2 style/consistency suggestions. There are no logic errors, broken links, incorrect parameter descriptions, or code changes of any kind. The documentation accurately describes the feature and its trade-offs. No files require special attention; minor example-value alignment between the config table and best-practices bullet in exact.mdx would improve clarity. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[ExactDeduplicationWorkflow] -->|reads blocks of input_blocksize| B[Input Blocks]
B -->|batched by identification_batchsize| C{Batch Ready?}
C -->|No — accumulate more blocks| B
C -->|Yes — batch full| D[Concatenate Blocks into Batch]
D --> E{Stage has read_and_insert_batch?}
E -->|Yes| F[read_and_insert_batch\nbatch-aware path]
E -->|No — fallback| G[Single-task insertion]
F --> H[Shuffler / GPU Operations]
G --> H
H --> I[ExactDuplicateIds output]
Reviews (2): Last reviewed commit: "docs: add release notes and docs for bat..." | Re-trigger Greptile |
| - **Xenna-managed CUDA devices**: Xenna now manages CUDA device visibility directly, replacing the previous Ray-managed approach. | ||
| - **Ray 2.54**: Updated Ray dependency to version 2.54 for compatibility with Cosmos-Xenna 0.2.0. | ||
|
|
||
| ### Batched Shuffle Insertion for Exact Deduplication (PR #1369) |
There was a problem hiding this comment.
I think of this more as an enhancement rather than what's new
1eb38bb to
5a6e2fa
Compare
Description
Adds 26.04 release notes and documentation for the batched shuffle insertion feature introduced in PR #1369. The release notes describe the new
identification_batchsizeparameter and batch-aware shuffle adapter. The exact dedup configuration table and performance best practices are updated accordingly.Usage
Checklist